Exploitation of structural sparsity in algorithmic differentiation

The background of this thesis is algorithmic differentiation (AD) [GW08] of in practice very computationally expensive vector functions F : R ⊇ D → R given as computer programs. Traditionally, most AD software1 provide forward and reverse modes of AD for calculating the Jacobian matrix ∇F (x) accurately at a given point x on some kind of internal representation of F kept on memory or hard disk. In fact, the storage is known to be the bottleneck of AD to handle larger problems efficiently in reverse mode. For instance, a tape is the internal representation of choice in the C++ operator overloading tool ADOL-C [GJM99] that presents an augmented version of F. Thus, ∇F can be obtained in forward and reverse fashion by an interpretative forward and reverse propagation of directional derivatives and adjoints [NMRC07] through the tape, respectively. The forward mode AD can be implemented very cheaply in terms of memory by single forward propagation of directional derivatives at runtime (tapeless in ADOL-C terminology). However, the reverse mode needs to store some data [HNP05] in the so-called forward sweep to allow the data flow reversal [Nau08] needed for backward propagation of adjoints. The latter is recently the focus of ongoing research activities of the AD community for m = 1 as a single application of reverse mode is enough to accumulate the gradient of F. To handle the memory bottleneck, checkpointing schedules e.g. revolve [GW00] have been developed for time-dependent problems. However, they require user’s knowledge in both the function F as well as the reverse mode AD. In this context, we aim to provide a tool, which minimizes non-AD experts effort in application of the reverse mode AD on their problems for large dimensions. Chapter 2 of this thesis is concerned with the accumulation of the Jacobian of F by the application of elimination techniques, which are very close to the Gaussian elimination performed in sparse LU factorization [PT08, FTPR04]. Thereby, we present algorithms that allow the application of elimination techniques [GN02] to the very large and sparse extended Jacobian of F being a lower triangular matrix of local partial derivative. However, the extended Jacobian is of quadratic memory complexity. Hence, compressed row storage [DER86] (CRS) representation is used to exploit its sparsity. This is done by first performing the so-called symbolic elimination step on the corresponding bit pattern of the extended Jacobian. This step predicts storage required for the statically allocated target CRS, which is used to accumulate the Jacobian of F at x. Nonetheless, the capability of the static CRS is also bounded by the memory consumption of the respective bit pattern even though the memory usage of CRS is considerably lower. To tackle this problem, elimination techniques are applied locally to the dense extended Jacobian (i.e without exploiting sparsity) and its CRS representation. Therefore, we keep track of the memory usage during the evaluation of F and apply elimination techniques whenever the memory bound is reached. The elimination is supposed to free memory enabling us to continue evaluating F. In fact, the evaluation of ∇F may require multiple evaluation and elimination steps. The former is supposed to provide the target data structure on which the latter is performed. We refer to this approach as iterative Jacobian accumulation. The implementations of the ideas above are provided in the C++ operator overloading tool DALG2 1Existing AD tools can be found on the community website www.autodiff.org. 2DALG stands for Derivative Accumulation for Large Graphs.